Chapter 8: NVIDIA Isaac Sim

"Isaac Sim brings photorealistic, physics-accurate simulation powered by RTX GPUs—bridging the reality gap like never before."

Table of Contents

1. Introduction to Isaac Sim
2. Installation and Setup
3. Isaac Sim Architecture
4. Creating Environments
5. Robot Import and Configuration
6. Synthetic Data Generation
7. ROS 2 Integration
8. Training with Isaac Sim
9. Performance Optimization

---

Introduction to Isaac Sim

NVIDIA Isaac Sim is a robotics simulation platform built on NVIDIA Omniverse, offering photorealistic rendering and accurate physics simulation powered by PhysX 5.

Why Isaac Sim?

Advantages:

Feature	Benefit
RTX Ray Tracing	Photorealistic rendering
PhysX 5	Accurate physics simulation
Synthetic Data	Perfect ground truth labels
GPU Acceleration	Fast parallel simulation
Domain Randomization	Improved sim-to-real transfer
ROS 2 Native	Seamless integration

Use Cases:

• 🤖 Humanoid robot development
• 🚗 Autonomous vehicle testing
• 🏭 Warehouse automation
• 🦾 Manipulation training
• 📸 Computer vision dataset generation

System Requirements

Minimum:

• NVIDIA RTX GPU (2070 or higher)
• 32GB RAM
• Ubuntu 20.04/22.04 or Windows 10/11
• 50GB storage

Recommended:

• NVIDIA RTX 3090/4090 or A6000
• 64GB+ RAM
• NVMe SSD
• Multi-GPU setup

---

Installation and Setup

Installing Isaac Sim

# Method 1: Omniverse Launcher
# Download from: https://www.nvidia.com/en-us/omniverse/
# Install Isaac Sim from the launcher

# Method 2: Docker (Recommended for production)
docker pull nvcr.io/nvidia/isaac-sim:2023.1.1

# Run Isaac Sim container
docker run --name isaac-sim --entrypoint bash -it \
  --gpus all \
  -e "ACCEPT_EULA=Y" \
  --rm \
  --network=host \
  -v ~/docker/isaac-sim/cache/kit:/isaac-sim/kit/cache:rw \
  -v ~/docker/isaac-sim/cache/ov:/root/.cache/ov:rw \
  -v ~/docker/isaac-sim/cache/pip:/root/.cache/pip:rw \
  -v ~/docker/isaac-sim/cache/glcache:/root/.cache/nvidia/GLCache:rw \
  -v ~/docker/isaac-sim/cache/computecache:/root/.nv/ComputeCache:rw \
  -v ~/docker/isaac-sim/logs:/root/.nvidia-omniverse/logs:rw \
  -v ~/docker/isaac-sim/data:/root/.local/share/ov/data:rw \
  nvcr.io/nvidia/isaac-sim:2023.1.1

Python Environment Setup

# Activate Isaac Sim Python environment
cd ~/.local/share/ov/pkg/isaac_sim-2023.1.1
source setup_conda_env.sh

# Install additional packages
pip install numpy==1.23.5
pip install torch torchvision

# Verify installation
python -c "from isaacsim import SimulationApp; print('Isaac Sim Ready!')"

ROS 2 Bridge Setup

# Install ROS 2 Bridge extension
# In Isaac Sim: Window → Extensions → ROS2 Bridge

# Or via command line
./isaac-sim.sh --ext-folder exts --enable omni.isaac.ros2_bridge

---

Isaac Sim Architecture

Core Components

┌───────────────────────────────────────────────┐
│           Isaac Sim Application               │
├───────────────────────────────────────────────┤
│  ┌─────────────┐  ┌──────────────────────┐   │
│  │   Python    │  │   Omniverse Kit      │   │
│  │   Scripts   │  │   (Extensions)       │   │
│  └─────────────┘  └──────────────────────┘   │
├───────────────────────────────────────────────┤
│  ┌─────────────┐  ┌──────────────────────┐   │
│  │   PhysX 5   │  │   RTX Rendering      │   │
│  │   Physics   │  │   Ray Tracing        │   │
│  └─────────────┘  └──────────────────────┘   │
├───────────────────────────────────────────────┤
│  ┌─────────────┐  ┌──────────────────────┐   │
│  │   USD       │  │   Replicator         │   │
│  │   (Scene)   │  │   (Synthetic Data)   │   │
│  └─────────────┘  └──────────────────────┘   │
└───────────────────────────────────────────────┘

USD (Universal Scene Description)

Isaac Sim uses USD for scene representation:

from pxr import Usd, UsdGeom, UsdPhysics

# Create USD stage
stage = Usd.Stage.CreateNew("robot_scene.usd")

# Create ground plane
plane = UsdGeom.Mesh.Define(stage, "/World/GroundPlane")
plane.CreatePointsAttr([(-10, -10, 0), (10, -10, 0), (10, 10, 0), (-10, 10, 0)])
plane.CreateFaceVertexCountsAttr([4])
plane.CreateFaceVertexIndicesAttr([0, 1, 2, 3])

# Add physics
collision_api = UsdPhysics.CollisionAPI.Apply(plane.GetPrim())

---

Creating Environments

Simple Environment

from isaacsim import SimulationApp

# Launch Isaac Sim
simulation_app = SimulationApp({"headless": False})

from omni.isaac.core import World
from omni.isaac.core.objects import DynamicCuboid, GroundPlane

# Create world
world = World()
world.scene.add_default_ground_plane()

# Add objects
cube = world.scene.add(
    DynamicCuboid(
        prim_path="/World/Cube",
        name="cube",
        position=[0, 0, 1.0],
        size=0.5,
        color=[1, 0, 0]
    )
)

# Run simulation
world.reset()

for i in range(1000):
    world.step(render=True)

simulation_app.close()

Warehouse Environment

from omni.isaac.core.utils.stage import add_reference_to_stage
from omni.isaac.core.utils.nucleus import get_assets_root_path

def create_warehouse():
    world = World()
    
    # Add warehouse asset
    assets_root_path = get_assets_root_path()
    warehouse_usd = f"{assets_root_path}/Isaac/Environments/Simple_Warehouse/warehouse.usd"
    
    add_reference_to_stage(usd_path=warehouse_usd, prim_path="/World/Warehouse")
    
    # Add shelves
    for i in range(5):
        for j in range(3):
            shelf_path = f"/World/Shelf_{i}_{j}"
            add_reference_to_stage(
                usd_path=f"{assets_root_path}/Isaac/Props/Shelves/Shelf.usd",
                prim_path=shelf_path
            )
            
            # Set position
            shelf = world.scene.add(
                XFormPrim(
                    prim_path=shelf_path,
                    position=[i * 2.0, j * 2.0, 0]
                )
            )
    
    return world

Outdoor Environment with Terrain

from omni.isaac.core.terrains import TerrainGenerator

def create_outdoor_scene():
    world = World()
    
    # Generate terrain
    terrain_generator = TerrainGenerator(
        num_rows=5,
        num_cols=5,
        horizontal_scale=0.25,
        vertical_scale=0.005,
        slope_threshold=0.75
    )
    
    terrain = terrain_generator.create_terrain()
    
    # Add to world
    world.scene.add(terrain)
    
    # Add sky
    add_reference_to_stage(
        usd_path="/Isaac/Environments/Outdoor/Sky.usd",
        prim_path="/World/Sky"
    )
    
    return world

---

Robot Import and Configuration

Importing URDF

from omni.isaac.core.utils.extensions import get_extension_path_from_name
from omni.isaac.core.utils.stage import add_reference_to_stage
from omni.isaac.core.robots import Robot
from omni.isaac.urdf import _urdf

def import_robot_from_urdf(urdf_path, robot_name="my_robot"):
    # Convert URDF to USD
    success, robot_prim_path = _urdf.import_urdf(
        urdf_path=urdf_path,
        prim_path=f"/World/{robot_name}",
        import_inertia_tensor=True,
        fix_base=False
    )
    
    if not success:
        raise Exception("Failed to import URDF")
    
    # Create robot object
    robot = Robot(prim_path=robot_prim_path, name=robot_name)
    
    return robot

# Usage
robot = import_robot_from_urdf("/path/to/robot.urdf", "mobile_robot")

Configuring Robot Articulation

from omni.isaac.core.articulations import ArticulationView

def configure_robot(robot_prim_path):
    # Get articulation
    articulation = ArticulationView(
        prim_paths_expr=robot_prim_path,
        name="robot_articulation"
    )
    
    # Set joint properties
    articulation.set_joint_position_targets(
        [0, 0, 0, 0, 0, 0],  # Joint positions
        joint_indices=[0, 1, 2, 3, 4, 5]
    )
    
    # Set joint stiffness
    articulation.set_gains(
        kps=[10000] * 6,
        kds=[1000] * 6
    )
    
    # Set joint limits
    articulation.set_joint_position_limits(
        lower_limits=[-3.14, -1.57, -1.57, -3.14, -1.57, -3.14],
        upper_limits=[3.14, 1.57, 1.57, 3.14, 1.57, 3.14]
    )
    
    return articulation

Adding Sensors to Robot

from omni.isaac.sensor import Camera, LidarRtx

def add_sensors_to_robot(robot_prim_path):
    # Add camera
    camera = Camera(
        prim_path=f"{robot_prim_path}/camera",
        position=[0.3, 0, 0.2],
        frequency=30,
        resolution=(1920, 1080),
        orientation=[0, 0, 0, 1]
    )
    
    camera.initialize()
    camera.add_motion_vectors_to_frame()
    camera.add_distance_to_image_plane_to_frame()
    
    # Add LiDAR
    lidar = LidarRtx(
        prim_path=f"{robot_prim_path}/lidar",
        config="Example_Rotary",
        position=[0.2, 0, 0.3],
        orientation=[0, 0, 0, 1]
    )
    
    lidar.initialize()
    
    return camera, lidar

---

Synthetic Data Generation

Replicator for Dataset Creation

import omni.replicator.core as rep

def generate_training_dataset():
    # Create camera
    camera = rep.create.camera(position=(3, 0, 2))
    
    # Create randomized objects
    with rep.new_layer():
        # Create objects to randomize
        shapes = rep.create.group([
            rep.create.sphere(scale=rep.distribution.uniform(0.5, 1.0)),
            rep.create.cube(scale=rep.distribution.uniform(0.5, 1.0)),
            rep.create.cylinder(scale=rep.distribution.uniform(0.5, 1.0))
        ])
        
        # Randomize position
        with shapes:
            rep.modify.pose(
                position=rep.distribution.uniform((-5, -5, 0), (5, 5, 3)),
                rotation=rep.distribution.uniform((-180, -180, -180), (180, 180, 180))
            )
        
        # Randomize appearance
        with shapes:
            rep.randomizer.color(
                colors=rep.distribution.choice([
                    (1, 0, 0),  # Red
                    (0, 1, 0),  # Green
                    (0, 0, 1),  # Blue
                    (1, 1, 0),  # Yellow
                ])
            )
    
    # Render and annotate
    render_product = rep.create.render_product(camera, (1024, 1024))
    
    # Add annotations
    writer = rep.WriterRegistry.get("BasicWriter")
    writer.initialize(
        output_dir="/tmp/dataset",
        rgb=True,
        bounding_box_2d_tight=True,
        semantic_segmentation=True,
        instance_segmentation=True,
        distance_to_camera=True,
        bounding_box_3d=True
    )
    
    writer.attach([render_product])
    
    # Run for 1000 frames
    with rep.trigger.on_frame(num_frames=1000):
        rep.randomizer.scatter_2d(shapes, surface_prims=["/World/Ground"])
    
    rep.orchestrator.run()

# Execute
generate_training_dataset()

Domain Randomization

import omni.replicator.core as rep

def apply_domain_randomization():
    # Randomize lighting
    def randomize_lighting():
        light = rep.get.prims(path_pattern="/World/Lights/*")
        with light:
            rep.modify.attribute(
                "intensity",
                rep.distribution.uniform(100, 5000)
            )
            rep.modify.attribute(
                "color",
                rep.distribution.uniform((0.8, 0.8, 0.8), (1, 1, 1))
            )
    
    # Randomize camera
    def randomize_camera():
        camera = rep.get.prims(path_pattern="/World/Camera")
        with camera:
            rep.modify.pose(
                position=rep.distribution.uniform(
                    (2, -2, 1),
                    (4, 2, 3)
                ),
                look_at="/World/Target"
            )
    
    # Randomize textures
    def randomize_textures():
        materials = rep.get.prims(semantics=[("class", "object")])
        with materials:
            rep.randomizer.materials(
                materials=rep.distribution.choice([
                    "/World/Materials/Metal",
                    "/World/Materials/Plastic",
                    "/World/Materials/Wood"
                ])
            )
    
    # Register randomizers
    rep.randomizer.register(randomize_lighting)
    rep.randomizer.register(randomize_camera)
    rep.randomizer.register(randomize_textures)
    
    # Trigger on each frame
    with rep.trigger.on_frame():
        rep.randomizer.randomize_lighting()
        rep.randomizer.randomize_camera()
        rep.randomizer.randomize_textures()

---

ROS 2 Integration

Publishing Robot State

from omni.isaac.core.utils.extensions import enable_extension

# Enable ROS2 bridge
enable_extension("omni.isaac.ros2_bridge")

import rclpy
from rclpy.node import Node
from sensor_msgs.msg import JointState

class IsaacRobotPublisher(Node):
    def __init__(self, robot):
        super().__init__('isaac_robot_publisher')
        self.robot = robot
        
        # Publisher
        self.joint_state_pub = self.create_publisher(
            JointState,
            'joint_states',
            10
        )
        
        # Timer
        self.timer = self.create_timer(0.01, self.publish_joint_states)
    
    def publish_joint_states(self):
        # Get joint positions from Isaac Sim
        joint_positions = self.robot.get_joint_positions()
        joint_velocities = self.robot.get_joint_velocities()
        
        # Create message
        msg = JointState()
        msg.header.stamp = self.get_clock().now().to_msg()
        msg.name = self.robot.joint_names
        msg.position = joint_positions.tolist()
        msg.velocity = joint_velocities.tolist()
        
        # Publish
        self.joint_state_pub.publish(msg)

Subscribing to Commands

from geometry_msgs.msg import Twist

class IsaacRobotController(Node):
    def __init__(self, robot):
        super().__init__('isaac_robot_controller')
        self.robot = robot
        
        # Subscriber
        self.cmd_vel_sub = self.create_subscription(
            Twist,
            'cmd_vel',
            self.cmd_vel_callback,
            10
        )
    
    def cmd_vel_callback(self, msg):
        # Apply velocity to robot in Isaac Sim
        linear_velocity = [msg.linear.x, msg.linear.y, 0]
        angular_velocity = [0, 0, msg.angular.z]
        
        self.robot.set_linear_velocity(linear_velocity)
        self.robot.set_angular_velocity(angular_velocity)

Complete ROS 2 Integration

from isaacsim import SimulationApp

simulation_app = SimulationApp({"headless": False})

import rclpy
from omni.isaac.core import World
from omni.isaac.core.robots import Robot

def main():
    # Initialize ROS 2
    rclpy.init()
    
    # Create world and robot
    world = World()
    robot = Robot(prim_path="/World/Robot", name="my_robot")
    world.scene.add(robot)
    
    # Create ROS 2 nodes
    publisher_node = IsaacRobotPublisher(robot)
    controller_node = IsaacRobotController(robot)
    
    # Reset world
    world.reset()
    
    # Simulation loop
    while simulation_app.is_running():
        # Step simulation
        world.step(render=True)
        
        # Spin ROS nodes
        rclpy.spin_once(publisher_node, timeout_sec=0)
        rclpy.spin_once(controller_node, timeout_sec=0)
    
    # Cleanup
    rclpy.shutdown()
    simulation_app.close()

if __name__ == '__main__':
    main()

---

Training with Isaac Sim

Reinforcement Learning Setup

from omni.isaac.gym.vec_env import VecEnvBase
import torch

class RobotReachEnv(VecEnvBase):
    def __init__(self, cfg, num_envs, device):
        super().__init__(cfg, num_envs, device)
        
        self.obs_dim = 10
        self.action_dim = 6
        
        self.create_envs()
    
    def create_envs(self):
        # Create multiple parallel environments
        for i in range(self.num_envs):
            env_path = f"/World/Env_{i}"
            
            # Add robot
            robot = self.world.scene.add(
                Robot(
                    prim_path=f"{env_path}/Robot",
                    name=f"robot_{i}"
                )
            )
            
            # Add target
            target = self.world.scene.add(
                DynamicSphere(
                    prim_path=f"{env_path}/Target",
                    position=[1, 0, 0.5],
                    radius=0.05,
                    color=[1, 0, 0]
                )
            )
    
    def get_observations(self):
        # Get robot state
        joint_positions = self.robots.get_joint_positions()
        joint_velocities = self.robots.get_joint_velocities()
        
        # Get end-effector position
        ee_positions = self.robots.get_end_effector_positions()
        
        # Get target positions
        target_positions = self.targets.get_world_poses()[0]
        
        # Compute observation
        obs = torch.cat([
            joint_positions,
            joint_velocities,
            ee_positions - target_positions
        ], dim=-1)
        
        return obs
    
    def pre_physics_step(self, actions):
        # Apply actions
        self.robots.set_joint_position_targets(actions)
    
    def post_physics_step(self):
        # Compute reward
        ee_positions = self.robots.get_end_effector_positions()
        target_positions = self.targets.get_world_poses()[0]
        
        distance = torch.norm(ee_positions - target_positions, dim=-1)
        reward = -distance
        
        # Check done
        done = distance < 0.05
        
        return reward, done

# Training loop
env = RobotReachEnv(cfg, num_envs=1024, device="cuda")

for episode in range(10000):
    obs = env.reset()
    
    for step in range(100):
        # Get action from policy
        action = policy(obs)
        
        # Step environment
        obs, reward, done = env.step(action)
        
        # Update policy
        policy.update(obs, action, reward)

---

Performance Optimization

Multi-GPU Training

import torch.distributed as dist

def setup_multi_gpu(rank, world_size):
    dist.init_process_group(
        backend='nccl',
        init_method='env://',
        world_size=world_size,
        rank=rank
    )
    
    torch.cuda.set_device(rank)

def train_multi_gpu(rank, world_size):
    setup_multi_gpu(rank, world_size)
    
    # Create environment on specific GPU
    env = RobotReachEnv(
        cfg,
        num_envs=256,  # Per GPU
        device=f"cuda:{rank}"
    )
    
    # Training loop
    # ...

# Launch
torch.multiprocessing.spawn(
    train_multi_gpu,
    args=(4,),  # 4 GPUs
    nprocs=4
)

Headless Mode

# Run without GUI for faster training
./isaac-sim.sh --headless --enable omni.isaac.gym

Performance Tips

Technique	Impact	Notes
Headless mode	High	2-3x speedup
Multi-GPU	High	Near-linear scaling
Reduce visual quality	Medium	Set low quality materials
Batch processing	High	Process multiple envs
Disable rendering	High	render=False during training

---

Summary

Isaac Sim provides cutting-edge simulation with photorealistic rendering and accurate physics, perfect for training robots before real-world deployment.

Key Takeaways:

1. ✅ RTX ray tracing for photorealism
2. ✅ PhysX 5 for accurate physics
3. ✅ Synthetic data generation with perfect labels
4. ✅ Native ROS 2 integration
5. ✅ Massively parallel training
6. ✅ Domain randomization for sim-to-real

Next Chapter:

Chapter 9 explores Isaac ROS, NVIDIA's hardware-accelerated perception stack for real-time AI on edge devices.

---

This chapter introduced NVIDIA Isaac Sim for photorealistic robot simulation and training. You're now ready to leverage GPU-accelerated simulation for your robotics projects.